Skip to content

enhance: batch broadcast tombstone GC - #33

Merged
sunby merged 1 commit into
codex/load-1m-segments-pr-stack-rebased-qv-workfrom
codex/batch-broadcast-tombstone-gc
Sep 11, 2026
Merged

enhance: batch broadcast tombstone GC#33
sunby merged 1 commit into
codex/load-1m-segments-pr-stack-rebased-qv-workfrom
codex/batch-broadcast-tombstone-gc

Conversation

@sunby

@sunby sunby commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Broadcast tombstone GC still performs one catalog deletion per task after #31, so a burst of completed broadcasts can leave a long deletion backlog. This change deletes eligible tombstones in batches bounded by metastore.maxEtcdTxnNum (default 64), preserving the existing count and lifetime retention rules.

Catalog I/O holds neither task nor manager locks. A batch advances the queue and marks its tasks DONE only after deletion succeeds; failures retain the batch for idempotent retry, including an ambiguous commit result. Late ACKs ignore TOMBSTONE/DONE tasks to prevent recreating deleted records, and manager shutdown cancels in-flight GC. Existing callback ordering and durable completion remain unchanged.

Validation on Go 1.26.6:

  • Catalog, broadcaster (including subpackages), and streamingcoord resource tests passed with -tags dynamic,test -gcflags="all=-N -l".
  • Targeted catalog and broadcaster concurrency tests passed 20 repetitions with -race and the same build tags/compiler flags.
  • Local embedded-etcd 3.7.1 validation passed: 129 exact-key deletions used 3 transactions, unrelated/prefix-neighbor keys survived, and retry/cancellation behaved as expected.
  • Focused lint passed with 0 issues; run_clang_format.sh completed. make lint-fix was run but remains blocked by the pre-existing undefined: mocks at internal/metastore/kv/querycoord/kv_catalog_test.go:372 in the base branch.

The regression tests cover retention boundaries, partial failure and retry, shutdown/recovery around deletion, late ACKs, and Load -> Release -> Load ordering through the real ACK scheduler with mocked business callbacks. A negative control removing the terminal-ACK guard fails on an unexpected catalog write.

A one-off local catalog benchmark on Go 1.26.6 / darwin-arm64 / Apple M5 deleted 4,096 pre-seeded records through the real etcd client:

Deletion mode Requests Elapsed
Individual 4,096 18.56 s
Batches of 64 64 0.41 s

These are single-sample local measurements, with seeding outside the timer; the benchmark and embedded-etcd validation used an external test overlay.

Cold-query latency and QPS still need a deployment-level rerun. TiKV ambiguous-result handling was traced through the existing wrapper but was not exercised against a live TiKV server. Sustained deletion deficits can still grow the pending queue.

Follow-up to #31.
Related: milvus-io/milvus#50549 (tombstone shutdown/recovery context).

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: sunby <sunbingyi1992@gmail.com>
@sunby
sunby merged commit cdd068f into codex/load-1m-segments-pr-stack-rebased-qv-work Sep 11, 2026
10 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant